Discussion herein is directed to video processing and, in particular, the area of filtering of video content with the goal of reducing or eliminating noise from frames of video. For example, it may be advantageous to reduce noise without adversely impacting or altering the video content itself, which may be difficult since identifying noise versus actual content is challenging. For example, not adversely impacting video content may include not blurring the content, not damaging the fine structures within the video content, preserving textures in video content as they are, and preserving edges in video content.
In the context of video processing, video generation typically begins with the capture of the output of a sensor/camera or scanning of film, which inherently results in capturing noise in addition to the intended video signal. The amount of captured noise in the content depends on many factors including the resolution used for capture of content, the sensitivity of the sensor, the ambient light, the presence of other electronics near the sensor, the type of film used and the exposure time, the bit-depth in analog-to-digitization process, and others. Also, the type of noise in the content may vary depending on the type of sensor and the capture process used, the type of film and the scanning process used, or the process of digitization of analog to digital signal. For instance, as a result of the aforementioned processes, digitized natural video source content may typically have Gaussian noise, salt-and-pepper noise, scene noise, film grain noise, or others.
One type of video content that is an exception in terms of not having noise is purely synthesized video content such as cartoons or animations. Sometimes noise is added to synthesized content to make the content appear more photo-realistic or natural. Furthermore, for the case of movie content generated from scanning of film, the choice of the type of film impacts the amount, strength and look-and-feel of noise such as fine grain noise, coarse grain noise, scattered grain noise, or dense grain noise etc. In fact, a movie producer/director may choose a type of film and exposure to capture a message or ambience (of an entire movie or a scene of a movie) they want to convey (e.g., whether the movie has a gritty look and feel by a higher strength, coarse grain noise or a mellow look and feel by a low strength or fine grain noise.
The preceding has been in the context of uncompressed original video content and not video content that has been compressed because, typically, denoising can be thought of as a preprocessing operation prior to encoding. However, in transcoding scenarios, the source video itself may be coded, albeit at high bitrates for ease of transmission or editing (this type of quality may be referred to as contribution quality). Furthermore, contribution quality video for distribution may need to be encoded to lower bit-rates (e.g., to distribution quality). In such cases, a denoiser may have to deal with additional sources of noise (e.g., from video coding itself) although if contribution bit-rates are high enough (such as in high quality transcoding) such noise may not be much of a problem. There are also other cases where somewhat lower quality transcoding may be needed such as for HD video from 15 Mbits/s to 5 Mbits/s. In such cases, the 15 Mbits/s coded video is likely to have coding artifacts. There are also cases where transcoding is needed simply to go from one video codec format to another and bit-rates may be lower and thus source video may have video coding artifacts.
For example, the artifacts introduced by video coding processes may also be considered as coding noise. Such artifacts include blockiness, blurriness, ringing, chroma bleeding, etc. The type and intensity of coding noise may depend on the codec standard (e.g. MPEG-2 or AVC), bit-rate, quality/speed tradeoffs in the codec, hardware/software codec, and the like.
Furthermore, in some contexts, noise filtering may be used as part of a preprocessor to make video content easier to encode. In this application, noise filtering may be used a bit more aggressively (with the consequence of some impact on fidelity of the content) to allow larger compression to be obtained by consequent encoding (e.g., by AVC, HEVC, etc.). In some applications, such as internet streaming of movies, during prime-time, etc., bandwidth that is normally available may have been cut-back drastically and such techniques may be necessary. Thus instead of encoding with higher quantizers in AVC or HEVC encoding, softening the content by denoise filtering prior to AVC or HEVC encoding may result in more graceful introduction of artifacts under the scenario of drastic cut-down in bandwidth. Noise filtering used in this manner is also useful for bit-rate control (e.g., in CBR systems) as it allows another control on bit-rate (e.g., varying the amount of filtering depending on resources available). Since the typical goal of any eventual fidelity based video coder is to (within constraints of available bit-rate) maintain as much fidelity of input video as possible, noisy video represents a problem since, for an encoder, it is hard to tell noise from content. Thus a video encoder has a much harder time dealing with noisy content as compared to video content with low or no noise.
As such, there is a continual demand for improved video denoising techniques, devices, systems, and the like. It is with respect to these and other considerations that the present improvements have been needed. Such improvements may become critical as the desire to provide high quality video denoising becomes more widespread.